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Executive  Summary 

The  Office  of  Naval  Research  (ONR)  has  funded  the  UCLA  National  Center  for 
Research  on  Evaluation,  Standards,  and  Student  Testing  (CRESST)  and  its 
subcontractor,  the  University  of  Southern  California  Behavioral  Technology 
Laboratories  (BTL),  to  conduct  research  on  the  integration  of  assessment  models  and 
tools  with  instructional  simulation  authoring  and  delivery  tools  designed  to  support 
Navy  and  Marine  Corps  distance  learning  (DL).  An  assessment  model  is  a  formal 
specification  of  tasks  and  measures  producing  data  that  can  be  used  as  evidence  to 
support  inferences  about  what  the  learner  knows.  For  this  project  the  assessment 
models  and  tools  provide  precision  measurement,  analysis,  and  diagnosis  of  learner 
performance  guiding  selection  and  delivery  of  remedial  training  through  the 
instructional  simulations.  ONR  is  also  funding  CHI  Systems,  Inc.  to  conduct  field- 
based  qualitative  research  on  Navy  and  Marine  Corps  DL  implementations  to 
develop  practical  guidelines  and  procedures  to  support  effective  DL  employment  in 
the  Navy  and  Marine  Corps. 
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The  initial  focus  of  these  research  efforts  was  USMC  marksmanship  training  at 
the  Camp  Lejeune  Weapons  Training  Battalion.  CHI  Systems,  Inc.  conducted  a 
qualitative  field-study-based  approach  to  assess  the  effectiveness  of  existing  DL 
while  the  CRESST/BTL  effort  used  the  marksmanship  training  context  to  explore 
new  options  for  DL  assessment  and  training.  Following  intensive  knowledge 
acquisition  to  build  an  appropriate  marksmanship  knowledge  base  to  guide 
assessment  development  and  scoring,  CRESST  created  and  pilot  tested  several  DL- 
delivered  assessment  tools  including  knowledge  mapping,  a  prior  knowledge 
questionnaire,  shot  group  depiction,  a  background  survey,  a  self-regulation  survey, 
and  an  assessment  of  rifle  marksmanship  measuring  Marines'  skill  at  identifying 
proper  and  improper  firing  positions.  Data  are  still  being  analyzed,  but  preliminary 
results  indicate  that  such  cognitive  measures  in  combination  with  background 
information  can  indeed  predict  shooter  performance  and  do  it  as  well  as  or  better 
than  far  more  expensive  and  time-consuming  measures.  CRESST's  data  recently 
collected  at  Camp  Lejeune's  Stone  Bay  Rifle  Range  indicate  that  a  combination  of 
background  variables  and  cognitive  measures  predict  qualification  scores  with  a 
multiple  regression  coefficient  of  .59.  Benchmarks  for  comparison  are  the  best  results 
obtained  using  performance  on  a  rifle  simulator  (Hagman,  1998),  which  show  a 
correlation  of  .69,  and  a  Marine's  most  recent  qualification  score,  which  shows  a 
correlation  of  .40. 

To  enable  use  of  assessment  results  in  selecting  and  delivering  remedial 
training,  BTL  and  CRESST  have  developed  and  prototyped  a  way  to  integrate 
assessment  results  and  interactive  distributed  training.  Simulations  and  simulation- 
centered  training  modules  for  marksmanship  training  were  developed  using  BTL's 
iRides  authoring  and  delivery  systems.  iRides  and  iRides  Author,  originally 
developed  with  ONR  support,  have  continued  to  be  enhanced  and  extended  in  the 
current  ONR-sponsored  project.  Simulation-based  training  modules  have  been 
developed  for  the  Battlesight  Zero  procedure  and  shot  group  analysis.  Additional 
modules  on  creating  the  proper  sight  picture  and  data  book  usage  are  under 
development. 

CHI's  field-based,  qualitative  study  of  Marine  Corps  marksmanship  training 
employed  extensive  interviews  with  trainers,  trainees,  and  administrators  to  provide 
an  overall  assessment  of  training  quality.  CHI  found  widespread  dissatisfaction  with 
current  Phase  I  training,  because  of  the  inconsistency  with  which  it  appears  to  be 
conducted.  Existing  advanced  technology  was  found  to  be  little  used.  Extensive 
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"down  time"  for  the  Indoor  Simulated  Marksmanship  Trainer  (ISMT)  since  its 
introduction  has  led  trainers  and  trainees  to  have  little  confidence  in  its  availability 
during  the  brief  Phase  I  training.  Problems  (whether  real  or  perceived)  of  skill 
transfer  from  the  simulator  to  the  real  firing  range  also  deterred  ISMT  usage.  In 
addition,  limited  number  of  available  computers  constrained  use  of  the  digitized 
marksmanship  ADL  course.  The  course  was  also  not  integrated  into  the  Phase  I 
curriculum  in  a  clear  way,  leading  trainers  away  from  its  use. 

The  combined  efforts  of  CHI,  CRESST,  and  BTL  provide  an  overall  picture  of 
the  current  state  of  DL  in  USMC  marksmanship  training  and  prototype  assessment 
and  instructional  simulation  tools  of  potential  benefit  to  USMC  marksmanship 
training  through  early  identification  and  remediation  of  Marines  likely  to  shoot 
poorly.  Early  identification  and  remediation  could  lead  to  savings  in  travel  cost,  time 
away  from  the  Marine's  home  unit,  the  coaches'  time  on  the  firing  line,  the  use  of 
firing  range  capacity,  and  the  cost  of  ammunition  and  targets.  In  addition  to  savings, 
early  identification  and  remediation  could  lead  to  higher  scores  overall,  fewer 
UNQs,  and  more  Experts.  To  realize  these  potential  benefits,  the  following  work 
should  be  completed.  The  first  two  tasks  would  require  transition  funding. 

•  Validation  of  the  assessments'  ability  to  go  beyond  prediction  of  overall 
shooting  performance  to  identify  specific  knowledge  gaps,  predict  the 
impact  of  the  missing  knowledge  on  shooting  performance,  and  identify  the 
remediation  required  to  fill  the  gap. 

•  Shrink-wrapping,  bullet-proofing,  and  validating  a  complete  assessment 
and  training  package  based  on  the  prototypes. 

•  Qualitative  studies  of  the  results  of  fielding  the  assessment  and  training 
package  produced  with  transition  funding,  including  analyses  of  how 
organizational  processes  adapt  (or  need  to  adapt)  to  make  effective  use  of 
DL  delivery,  and  guidelines  and  a  supporting  case  study  helping  the  Navy 
(via  NETC)  and  Marine  Corps  (via  the  DLC)  better  design  and  employ  DL. 

In  addition,  the  following  work  might  be  considered  for  additional  R&D 
funding: 


•  Integrating  assessment  and  training  with  the  forthcoming  Marine  LOMAH 
(Location  Of  Miss  And  Hit)  system.  This  may  make  it  possible  to  assess 
shooting  performance  directly  and  automatically,  and  to  select  appropriate 
training  modules  based  on  that  performance. 


Assessment  Tools  and  Instructional  Simulations  for  USMC  Marksmanship  Training 


Vll 


•  Hand-held  assessment  and  training  modules.  These  could  be  used  to  teach 
data  book  activities  in  the  context  of  real  shooting  environments,  with 
automatic  evaluation  and  feedback.  Advice  could  be  provided  on  the  firing 
line,  and  instructors  could  be  automatically  alerted  about  problems.  If 
integrated  with  the  LOMAH  system,  these  hand-held  devices  could  also 
provide  feedback  and  advice  about  actual  shooting  performance. 
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Introduction 

The  UCLA  National  Center  for  Research  on  Evaluation,  Standards,  and  Student 
Testing  (CRESST)  and  its  subcontractor,  the  University  of  Southern  California 
Behavioral  Technology  Laboratories  (BTL),  are  under  contract  to  the  Office  of  Naval 
Research  (ONR)  to  conduct  research  on  assessment  models  and  tools  integrated 
with  instructional  simulation  authoring  and  delivery  tools  designed  to  support 
Navy  and  Marine  Corps  distance  learning  (DL).  The  project  is  called  Knowledge, 
Models  and  Tools  to  Improve  the  Effectiveness  of  Naval  Distance  Learning,  or  KMT. 
CRESST  is  responsible  for  assessment  models  and  tools.  An  assessment  model  is  a 
formal  specification  of  tasks  and  measures  producing  data  that  can  be  used  as 
evidence  to  support  inferences  about  what  the  learner  knows.  BTL  is  responsible  for 
instructional  simulation  authoring  and  delivery  tools.  For  this  project  the  assessment 
models  and  tools  provide  precision  measurement,  analysis,  and  diagnosis  of  learner 
performance  guiding  selection  and  delivery  of  remedial  training  through  the 
instructional  simulations.  The  approach  to  conducting  KMT  research  has  been  to 
develop  and  test  tools  designed  to  address  the  assessment  and  training  requirements 
posed  by  real  Navy  and  Marine  Corps  training  applications.  The  first  such 
application  is  USMC  marksmanship  training. 


Assessment  Tools  and  Instructional  Simulations  for  USMC  Marksmanship  Training 


2 


ONR  has  also  funded  CHI  Systems,  Inc.  to  conduct  ethnographic  research  on 
Navy  and  Marine  Corps  DL  implementations,  the  purpose  being  to  document  and 
evaluate  the  transformation  process  from  traditional  to  DL  training  and  identify 
what  works,  what  does  not  work,  causes  of  problems,  and  successful  solutions  to 
problems  in  order  to  develop  guidelines  for  DL  implementation.  In  FY02,  CHI  was  a 
subcontractor  to  both  CRESST  and  the  Naval  Air  Systems  Command,  Training 
Systems  Division  (NAVAIR  TSD).  In  FY03,  the  company's  work  on  DL 
implementation  continues  to  be  funded  by  ONR,  but  the  contract  is  managed  solely 
by  NAVAIR  TSD. 

This  document  provides  a  summary  of  work  completed  to  date  on  the  USMC 
marksmanship  application.  It  begins  with  an  overview  of  the  current  marksmanship 
training  process  and  a  description  of  the  concept  for  using  assessment  and 
simulation  tools  to  meet  marksmanship  training  requirements.  Knowledge 
acquisition,  assessment  development,  and  pilot  testing  are  then  described,  followed 
by  an  overview  of  BTL's  work  in  using  instructional  simulation  authoring  and 
delivery  tools  to  develop  simulations  providing  remedial  training  in  marksmanship 
fundamentals.  The  document  ends  with  a  summary  of  CHI's  work  toward 
developing  a  baseline  description  of  the  existing  marksmanship  training  program, 
which  will  enable  documentation  of  transformations  that  may  occur  with  adoption 
of  a  DL  approach  to  marksmanship  training,  possibly  but  not  necessarily  including 
the  KMT  assessment  tools  and  instructional  simulations. 

The  Current  Marksmanship  Training  Process 

KMT  project  work  on  marksmanship  training  was  conducted  at  the  Stone  Bay 
Rifle  Range,  Camp  Lejeune,  North  Carolina.  Stone  Bay  marksmanship  training 
spans  one-and-a-half  weeks  and  is  divided  into  three  Phases.  Phase  I  provides  the 
Marines  with  instruction  on  the  fundamentals  of  marksmanship.  Phase  II  is  actual 
live  firing  of  the  weapon  on  the  Known-Distance  (KD)  course.  Phase  III  is  field 
firing. 

Phase  I  has  two  components:  classroom  instruction  and  snapping-in.  During 
the  classroom  instruction.  Marines  are  given  a  review  of  the  fundamentals  of 
marksmanship.  For  all  Base  Marines  and  some  Group  Marines,  Phase  I  is  conducted 
at  Stone  Bay.  Marines  who  do  not  complete  Phase  I  at  Stone  Bay  conduct  it  at  their 
individual  units  with  their  Primary  Marksmanship  Instructors  (PMI). 


Assessment  Tools  and  Instructional  Simulations  for  USMC  Marksmanship  Training 


3 


Annual  qualification  at  Stone  Bay  spans  the  course  of  a  week  and  a  half.  Phase  I 
begins  on  Wednesday  of  the  first  week.  Marines  are  given  12-15  hours  of  classroom 
instruction  over  the  course  of  two  days.  The  courses  are  taught  by  BMTU  (Base 
Marine  Training  Unit)  for  Base  Marines  and  FSSGMTU  (Force  Service  Support 
Group  Marine  Training  Unit)  for  the  Group  Marines.  FSSGMTU  gives  their  Marines 
an  entrance  survey  consisting  of  five  questions  regarding  their  previous  annual 
qualification.  For  example.  Marines  are  asked  to  provide  their  previous  shooting 
score  and  any  perceived  difficulties  they  may  have  regarding  rifle  marksmanship. 
FSSGMTU  also  gives  a  Phase  I  final  exam  at  the  end  of  classroom  instruction.  The 
coaches  use  these  tools  as  one  means  of  assessing  Marines  who  may  not  have 
grasped  the  fundamentals. 

Each  day  after  classroom  instruction,  Marines  move  to  the  grass  where  they 
spend  approximately  three  hours  "snapping-in,"  which  consists  of  practice  getting 
in  and  out  of  positions,  sighting  in,  and  running  through  a  mock  KD  course.  One 
coach  is  available  per  5  to  10  Marines  to  provide  individual  assistance. 

Marines  who  complete  Phase  I  at  their  individual  units  are  required  to  have 
classroom  instruction  and  to  spend  time  snapping-in.  Upon  arriving  at  Stone  Bay, 
they  are  required  to  submit  a  letter  from  their  unit  commander  verifying  their 
completion  of  Phase  I  prior  to  beginning  Phase  II. 

On  Friday  of  the  first  week,  all  qualifying  Marines  arrive  at  Stone  Bay  for  their 
grouping  and  battlesight  zero  exercises.  Marines  are  on  deck  by  0630  and  are  done 
by  1230.  Phase  II  begins  on  the  following  Monday.  On  Monday,  no  qualification 
takes  place.  Marines  shoot  the  KD  course  with  coaches  on  the  firing  line  assisting  the 
Marines.  The  coaches'  priority  is  safety,  not  instruction,  and  they  are  not  able  to 
provide  much  individualized  attention.  The  ratio  of  coaches  to  Marines  is  about  1:4. 
Starting  Tuesday,  Marines  have  the  option  of  either  attempting  qualification  or 
continuing  to  shoot  the  KD  course  for  practice.  Those  who  qualify  may  return  to 
their  units,  but  must  return  on  Friday  for  Phase  III.  Marines  who  attempt  to  qualify 
but  shoot  below  a  25  are  classified  as  unqualified  and  must  return  on  Wednesday  for 
training  and  attempt  qualification  on  Thursday.  On  Wednesday,  those  who  did  not 
attempt  qualification  on  Tuesday  may  attempt  qualification.  On  Thursday,  all 
Marines  must  attempt  qualification  or  reattempt  qualification  if  they  did  not  qualify 
on  either  of  the  previous  two  days.  Currently,  Marines  who  do  not  qualify  by 
Thursday  are  given  the  option  of  returning  to  Stone  Bay  to  repeat  the  entire  process 
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at  a  later  date,  or  to  reattempt  qualification  Thursday  afternoon  after  additional 
training. 

Phase  III  is  on  Friday  of  the  last  week.  All  Marines  return  to  Stone  Bay  and 
engage  in  field  firing.  Field  firing  is  the  part  of  annual  qualification  when  combat 
actions  are  most  closely  mirrored.  For  example.  Marines  wear  bulletproof-type 
jackets,  helmets,  and  gas  masks  while  shooting.  Likewise,  the  targets  move  from 
side  to  side  and  multiple  targets  are  presented. 

The  Concept:  Using  Assessment  to  Increase  Productivity  and  Effectiveness 

In  order  to  reduce  the  number  of  Marines  in  the  Unqualified  category  (UNQ) 
and  increase  the  number  attaining  the  Expert  rating,  Lt.  Col.  Carl  Shelton  of  the 
Camp  Lejeune  Weapons  Training  Battalion  (WTBN)  developed  a  concept  called 
UNQ  to  Expert,  abbreviated  as  U  to  E  or  "U-E,"  and  now  known  as  "Huey." 
Initially,  the  idea  was  that  assessment  tools  and  simulations  could  be  used  to 
provide  remedial  training  to  Marines  after  they  failed  to  qualify.  WTBN  and  KMT 
staff  soon  realized,  however,  that  there  was  much  greater  potential  for  time  and  cost 
savings  if  online  assessments  administered  before  Marines  reached  the  firing  line  — 
and  even  better,  before  they  reached  Camp  Lejeune— could  identify  Marines  likely 
to  shoot  poorly  and  identify  specific  needs  for  remedial  training  that  could  be  met 
by  online  instructional  simulation,  the  Indoor  Simulated  Markmanship  Trainer 
(ISMT),  and/or  the  Marine  Corps  Institute  (MCI)  Rifle  Marksmanship  online  course. 

The  potential  for  time  and  cost  savings  is  there:  Early  identification  and 
remediation  could  lead  to  savings  in  travel  cost,  time  away  from  the  Marine's  home 
unit,  the  coaches'  time  on  the  firing  line,  the  use  of  firing  range  capacity,  and  the  cost 
of  ammunition  and  targets.  In  addition  to  savings,  early  identification  and 
remediation  could  lead  to  higher  scores  overall,  fewer  UNQs,  and  more  Experts.  But 
the  potential  for  savings  and  enhanced  shooting  performance  depends  on  the  ability 
to  identify  likely  poor  shooters  and  their  needs  for  remedial  training  using  online 
assessments.  Shooting  is  obviously  primarily  a  motor  skill,  and  it  is  difficult  if  not 
impossible  to  assess  a  motor  skill  in  a  DL  environment.  It  is  possible,  however,  to 
assess  knowledge— facts,  concepts,  and  their  relationships  —  in  a  DL  environment.  If 
it  could  be  shown  that  online  assessments  of  knowledge  could  predict  shooting 
performance  and  identify  the  knowledge  gap  driving  the  prediction,  then  Huey 
could  work. 
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Knowledge  Acquisition,  Assessment  Development,  and  Pilot  Testing 

Assessment  development  must  be  based  on  a  solid  understanding  of  the 
domain  to  be  assessed.  CRESST  researchers  visited  the  Stone  Bay  Rifle  Range  at 
Camp  Lejeune  five  times  between  April  and  December  of  2002  to  acquire  such 
knowledge,  which  included  the  collection  of  rifle  marksmanship  concepts  and 
fundamentals  through  an  intensive  review  of  marksmanship  manuals  and  training 
materials  (see  references  USMC,  1992;  USMC,  1999;  and  USMC,  2001).  CRESST  also 
conducted  extensive  interviews  with  WTBN  staff  regarding  how  best  to  assess  the 
cognitive  aspect  of  shooting  to  predict  shooting  performance. 

Early  in  knowledge  acquisition  it  became  clear  that  high-performing  marksmen 
know  much  more  about  the  fundamentals  of  rifle  marksmanship  than  poor  or 
average  marksmen  and  use  this  knowledge  to  improve  their  shooting.  High- 
performing  marksmen  know  the  fundamentals,  the  relationships  among 
fundamentals,  the  effects  of  fundamentals  on  shooting  performance,  how  to 
recognize  problems  in  applying  fundamentals,  and  how  to  fix  problems  in  applying 
fundamentals.  They  use  their  data  books  as  diagnostic  tools.  They  use  shot  patterns 
and  the  discrepancy  between  the  called  shot  location  and  the  actual  shot  location  to 
identify  and  correct  problems  in  applying  fundamentals  such  as  breath  control, 
trigger  control,  and  sight  adjustment  compensation.  Better  shooters  are  able  to 
describe  the  relationships  of  elements  of  shooting  positions.  For  example,  they  know 
that  the  grip  of  the  firing  hand  affects  trigger  control,  and  that  stock  weld  placement 
affects  eye  relief. 

The  knowledge  acquired  from  reviews  of  manuals  and  training  materials  was 
organized  into  a  network  of  relationships  and  stored  in  an  online  knowledge  base. 
The  knowledge  base  is  used  to  generate  and  score  assessments,  and  will  be  used  to 
identify  specific  remedial  training  based  on  knowledge  gaps  revealed  by  assessment 
results.  Screens  providing  views  into  the  knowledge  base  are  shown  in  the 
Powerpoint  file  stored  on  the  CD  delivered  at  the  December  19,  2002,  CRESST/ BTL 
briefing  in  Quantico. 

The  marksmanship  knowledge  base  defines  what  can  be  known  about 
marksmanship:  marksmanship  fundamentals,  the  relationships  among 

fundamentals,  the  effects  of  fundamentals  on  shooting  performance,  how  to 
recognize  problems  in  applying  fundamentals,  and  how  to  fix  problems  in  applying 
fundamentals.  The  purpose  of  assessment  in  the  Marine  marksmanship  application 
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is  to  determine  what  is  known  by  an  individual  Marine  in  order  to  identify  Marines 
who  may  shoot  poorly  and  identify  specific  needs  for  remedial  training. 

Assessments  are  designed  to  elicit  learner  behaviors,  producing  data  that  can 
be  used  as  evidence  to  support  inferences  about  what  the  learner  knows  (Pellegrino, 
Chudowsky,  and  Glaser,  2001).  Data  may  include  marks  on  answer  sheets,  graphic 
representations  or  "concept  maps"  showing  relationships  among  fundamentals  and 
relationships  of  fundamentals  to  shot  placement,  or  a  response  to  a  problem  posed 
by  an  online  simulation.  To  be  used  as  evidence,  it  must  be  possible  to  interpret  the 
data  in  terms  of  their  relevance  to  what  the  Marine  knows  (and  does  not  know)  that 
is  important  to  shooting  performance.  This  requires  that  assessment  tasks  be 
designed  that  will  provide  data  that  can  be  linked  to  the  cognitive  model  to  support 
inferences  about  the  Marine's  knowledge  of  marksmanship  fundamentals,  the 
relationship  among  fundamentals,  the  effect  of  fundamentals  on  shooting 
performance,  how  to  recognize  problems  in  applying  fundamentals,  and  how  to  fix 
problems  in  applying  fundamentals.  But  it  is  not  enough  to  simply  design 
assessment  tasks.  The  assessments  must  also  be  validated  by  using  them  to  collect 
data  and  determining  whether  the  data  can  be  used  to  predict  shooting  performance 
as  indicated  by  the  knowledge  base. 

A  focus  on  the  cognitive  aspects  of  rifle  marksmanship  is  unusual  in  research 
on  largely  psychomotor  skills.  Prior  research  on  other  sports  recognizes  the 
existence  of  cognitive  aspects  as  well  as  motor  skills  (McPherson  &  French,  1991; 
Thomas,  French,  &  Humphries,  1986).  Most  marksmanship  research  has  ignored  the 
cognitive  dimension,  focusing  on  using  scores  from  rifle  simulators  to  predict 
qualification  scores  (e.g.,  Evans,  Dyer,  &  Hagman,  2000;  Hagman,  1998;  Schendel, 
Heller,  Finley,  &  Hawley,  1985;  Smith  &  Hagman,  2000;  Torre,  Maxey,  &  Piper, 
1987),  as  well  as  examining  the  relationship  between  background  variables  and 
shooting  performance  (e.g.,  Carey,  1990).  In  general,  correlations  between  device-fire 
scores  (from  rifle  simulators)  and  record-fire  scores  range  from  .40  to  .70,  which  we 
considered  the  benchmarks  for  our  research. 

To  gather  information  about  the  quality  of  assessments,  CRESST  conducted 
two  pilot  studies  and  one  main  study.  The  purpose  of  the  first  pilot  study  was  to 
gather  preliminary  data  on  the  relationships  between  knowledge  of  rifle 
marksmanship  and  shooting  performance.  The  purpose  of  the  second  pilot  study 
was  to  test  the  online  versions  of  our  software  in  the  target  environment.  We  wanted 
to  identify,  prior  to  the  main  study,  any  technical  issues  as  well  as  gather 
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information  on  the  usability  of  our  assessments.  The  main  study  was  the  test  of  our 
assessments  in  the  target  mode  of  delivery  (online)  using  as  controlled  procedures 
as  feasible. 

Study  A— Pilot  Test  of  Measures  0une  2002) 

During  this  pilot  test  we  interviewed  the  coaches  and  command  staff  at  Stone 
Bay  to  gather  responses  on  our  measures,  correlated  scores  on  our  measures  with 
live-fire  scores,  and  examined  the  strength  of  the  relationship.  We  gathered  input 
from  the  Stone  Bay  command  staff  and  coaches,  practiced  firing  an  M16A2, 
interviewed  coaches,  and  observed  firing  procedures. 

Measures.  The  following  measures  were  administered  to  approximately  60 
Marines: 

1.  Knowledge  map.  Knowledge  mapping  was  intended  to  measure  in  general 
a  Marine's  knowledge  of  rifle  marksmanship  and  in  particular,  cause-effect 
relations  among  different  aspects  of  marksmanship.  The  task  required 
Marines  to  graphically  depict  their  understanding  of  rifle  marksmanship  in 
terms  of  a  network.  The  nodes  in  the  network  represented  concepts,  and 
labeled  links  between  nodes  represented  the  relationships  between 
concepts.  The  sets  of  concept  and  link  labels  were  provided  to  Marines. 

2.  Prior  knowledge.  The  intent  of  the  prior  knowledge  task  was  to  survey 
Marines'  knowledge  of  rifle  marksmanship.  Marines  were  given  20  terms 
and  asked  to  provide  a  short  written  response  that  could  define  the  term  or 
explain  the  significance  of  the  term. 

3.  Shot-to-shot  explanation.  The  intent  of  this  task  was  to  measure  a  Marine's 
reasoning  from  shot  to  shot.  This  measure  was  an  attempt  to  mimic  the 
cognitive  demands  of  what  a  Marine  would  experience  on  the  firing  line 
under  slow-fire  conditions  (i.e.,  fire  a  round,  observe  the  strike  and  evaluate 
possible  causes  that  could  affect  the  strike).  Five  frames  were  shown  to  the 
Marine.  Each  successive  frame  showed  an  additional  bullet  strike.  For  each 
frame,  the  Marine  was  given  a  list  of  18  possible  reasons  for  the  shot 
pattern.  The  Marine  was  required  to  check  off  all  possible  causes  that  could 
give  rise  to  the  observed  shot  pattern. 

4.  Shot  group  prediction  task.  The  intent  of  this  task  was  to  measure  the  extent 
to  which  a  Marine  could  explain  a  shot  pattern  and  explain  adjustments 
required  to  go  from  one  shot  group  to  another  shot  group. 

5.  Shot  group  depiction  task.  The  intent  of  this  task  was  to  measure  Marines' 
knowledge  of  shot  groups  associated  with  common  shooter  problems. 
Marines  were  required  to  draw  a  5-shot  group  for  problems  with  breathing, 
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sight  adjustment,  trigger  control,  flinching,  bucking,  and  focusing  on  the 
target, 

6.  Background  survey.  The  intent  of  this  task  was  to  be  able  to  characterize  the 
sample  and  identify  background  characteristics  that  could  differentiate 
high-  from  low-performing  shooters.  The  kind  of  information  gathered 
included  unit  information,  prior  shooting  experience  and  scores,  and 
marksmanship  training  experience. 

Results.  Analyses  of  our  preliminary  measures  suggested  a  relationship 
between  Marines'  self-reported  qualification  scores  and  knowledge  of 
marksmanship.  Based  on  this  particular  sample  of  Marines,  we  found  that  Marines' 
self-reported  qualification  score  was  related  to  the  quality  of  their  knowledge  map. 
Self-reported  qualification  scores  were  best  predicted  by  whether  the  Marine  took 
the  coach's  course  and  the  frequency  of  shooting  outside  of  their  USMC  duties 
(multiple  R  =  .62,  p  <  .05),  followed  by  the  extent  to  which  they  were  able  to  depict 
shot  groups  associated  with  common  shooter  errors  (multiple  R  =  .55,  p  <  .05). 
Quality  of  the  knowledge  map  was  also  related  to  qualification  scores,  although 
marginally  statistically  significant  (r  =  .36,  p  =  .08,  n  =  24).  Marines'  self-reported 
previous  year's  qualification  score  was  also  related  to  self-reported  qualification  score 
(r  =  .38,  p  <  .05). 

Limitations.  Note  that  these  results  were  based  on  a  sample  of  Marines  who 
were  administered  paper  measures  under  very  poor  administration  conditions. 
Marines  were  sitting  on  bleachers  on  the  firing  range,  under  windy  conditions.  Also, 
the  measures  were  exploratory  and  were  our  first  attempt  at  measuring  knowledge 
of  rifle  marksmanship.  Thus,  these  results  are  at  best  only  suggestive  of  a 
relationship  between  shooting  skill  and  knowledge. 

Study  B— Pilot  Test  of  Computer  Format  of  Measures  (November  2002) 

During  this  pilot  test  we  carried  out  a  technical  dry  run  of  our  main  study  and 
tested  the  usability  of  our  online  assessments  with  Marines.  We  dropped  measures 
from  Study  A  that  did  not  appear  promising,  revised  promising  measures,  and 
converted  the  paper  measures  to  an  online  format.  During  the  administration  of  our 
measures,  numerous  technical  issues  arose.  The  most  serious  issue  was  the  severe 
security  restrictions,  which  prevented  the  administration  of  our  QuickTime-based 
assessments. 
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Measures.  The  following  measures  were  administered  to  approximately  100 
Marines: 

1.  Knowledge  map.  We  refined  the  knowledge  map  set  of  terms  and  links  and 
developed  an  online  version  that  Marines  could  use  to  draw  their  maps.  We 
also  administered  a  paper  version. 

2.  Prior  knowledge.  We  converted  the  prior  knowledge  measure  from  a  short- 
answer  format  to  a  selected-response  format.  We  developed  items  for  the 
following  areas  and  adopted  several  items  from  a  Stone  Bay/FSSG- 
developed  test:  sight  picture,  sight  adjustment,  sight  alignment,  weapons 
safety,  breathing,  trigger  control,  stock  weld,  eye  relief,  bone  support,  firing 
hand  placement,  follow-through,  muscle  relaxation,  forward  hand 
placement,  grip  of  firing  hand,  and  muscular  tension. 

3.  Shot  group  depiction  task.This  was  like  the  shot  group  depiction  task  used 
in  Study  A,  except  that  drawing  a  shot  group  for  trigger  control  was 
dropped. 

4.  Self-regulation  survey.  The  intent  of  this  task  was  to  measure  Marines'  use 
of  self-regulation  strategies  with  respect  to  classroom-based  learning  (e.g., 
planning,  checking  work). 

5.  Background  survey.  This  was  like  the  background  survey  used  in  Study  A, 
with  some  revised  items. 

Results.  In  terms  of  relationships  between  our  assessments  and  qualification 
score,  we  again  found  relationships  that  suggested  a  link  between  knowledge  of  rifle 
marksmanship  and  self-reported  qualification  score.  For  example,  Marines'  self- 
reported  qualification  score  was  related  to  performance  on  our  prior  knowledge 
measure  (r  =  .21,  p  <  .05);  if  they  rated  their  job  as  non-ground  combat  (r  =  .29,  p  < 
.01);  the  number  of  months  since  their  last  Phase  I  training  ( r  =  -.25,  p  <  .05);  and 
their  self-reported  most  recent  qualification  score  (r  =  .36,  p  <  .01). 

With  respect  to  performance  on  our  assessments,  we  found  that  Marines  scored 
higher  on  our  prior  knowledge  measure  (a)  if  they  had  Phase  I  training  (r  =  .20,  p  < 
.05);  (b)  the  more  recently  (in  months)  they  had  Phase  I  training  (r  =  -.24,  p  <  .05); 
and  (c)  the  higher  their  self-reported  most  recent  qualification  score  (r  =  .23,  p  <  .05). 
Also,  Marines  tended  to  perform  higher  on  the  shot  group  task  (a)  if  they  rated  their 
job  as  non-ground  combat  (r  =  .21,  p  <  .05);  and  (b)  the  more  hours  they  reported 
shooting  as  part  of  their  USMC  duties  (r  =  .30,  p  <  .05). 

Finally,  the  best  predictor  of  self-reported  qualification  scores  was  from  the 
following  set  of  variables:  (a)  average  of  scores  on  the  planning  and  checking  scales 
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on  the  self-regulation  survey,  (b)  number  of  months  since  the  Marine's  last  Phase  I 
training,  (c)  score  on  the  prior  knowledge  measure,  (d)  self-reported  most  recent 
qualification  score,  and  (e)  combat  status  of  the  Marine's  job.  The  resulting  multiple 
R  for  this  sample  was  .53. 

Limitations.  As  with  the  first  pilot  study,  the  results  related  to  our  assessments 
need  to  be  interpreted  with  caution.  These  data  are  from  Marines  who  took  our 
measures  online  and  on  paper.  Further,  the  Marines  who  received  the  online 
assessments  experienced  technical  problems  with  the  software  (e.g.,  slow-loading  or 
non-loading  Web  pages;  very  slow  system  performance). 

Given  this  caution,  the  results  again  suggest  a  knowledge  component  to 
shooting  performance.  As  in  Study  A,  self-reported  qualification  scores  were  related 
significantly  to  measures  of  knowledge  of  the  fundamentals,  although  the 
magnitude  of  the  relationship  was  low.  Evidence  that  our  prior  knowledge  measure 
was  tapping  knowledge  of  rifle  marksmanship  is  seen  in  the  relationship  with  the 
Phase  I  training  variables  (i.e.,  higher  scores  on  the  prior  knowledge  measure  if  they 
had  Phase  I  training  and  the  more  recent  the  Phase  I  training).  Finally,  the  regression 
equation  also  supports  the  interpretation  that  there  is  a  knowledge  component  to 
shooting  performance.  Prior  knowledge  scores  and  recency  of  Phase  I  training 
improves  the  predictability  of  shooting  performance  above  experience  variables 
alone. 

Study  C— Pilot  Test  of  Complete  Set  of  Online  Measures  (December  2002) 

The  study  conducted  in  December  2002  was  intended  to  examine  the  extent  to 
which  our  online  assessments  can  predict  shooting  performance.  The  full  set  of 
measures  were  administered  online,  and  the  current  and  most  recent  qualification 
scores  were  gathered  from  the  Stone  Bay  database,  not  Marines'  self-reports. 
CRESST  researchers  also  observed  the  pit  verification  process  and  Stone  Bay  also 
provided  verifiers  to  ensure  the  Marines'  bullet  strikes  were  accurately  recorded  on 
the  score  cards.  Technical  problems  with  the  online  delivery  were  resolved  and  the 
online  administration  during  the  data  collection  was  problem  free.  Training  on  our 
assessment  tasks  was  standardized  to  ensure  that  each  Marine  had  the  same 
training.  Further,  five  CRESST  researchers  and  programmers  proctored  the  data 
collection. 

Measures.  The  following  measures  were  administered  to  Marines: 
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1.  Knowledge  map.  We  used  the  same  measure  from  Study  B. 

2.  Prior  knowledge.  We  used  the  same  measure  from  Study  B  with  slight 
modifications  to  wording. 

3.  Shot  group  depiction  task.  We  used  the  same  measure  from  Study  B. 

4.  Self-regulation  survey.  We  used  the  same  measure  from  Study  B. 

5.  Background  survey.  We  used  the  same  measure  from  Study  B  with  slight 
modifications  to  wording. 

6.  Assessment  of  Rifle  Marksmanship  Skills  (ARMS).  This  task  was  intended 
to  measure  Marines'  skill  at  identifying  proper  and  improper  firing 
positions.  The  shooter  was  shown  in  QuickTime  VR,  and  Marines  could 
rotate  the  image  to  view  the  shooter  from  different  angles.  Marines  were 
asked  to  judge  whether  the  shooter's  position  was  proper  or  improper  on 
the  following  elements:  placement  of  firing  hand,  placement  of  forward 
hand,  elbow  placement,  stock  weld,  breath  control,  rifle  butt  in  pocket  of 
shoulder,  leg  placement,  feet  placement,  body  placement,  and  overall 
position. 

7.  Evaluation.  A  short  survey  asked  Marines  how  much  they  knew  of  rifle 
marksmanship,  their  perceived  importance  of  knowledge  of  marksmanship 
to  shooting  performance,  and  the  difficulty  of  our  assessments. 

Results.  In  terms  of  relationships  between  our  assessments  and  qualification 
score,  we  again  found  relationships  that  suggested  a  link  between  knowledge  of  rifle 
marksmanship  and  official  qualification  score.  For  example.  Marines'  official 
qualification  score  was  related  to  their  performance  on  our  prior  knowledge 
measure  ( r  =  .31,  p  <  .01),  shot  group  depiction  measure  ( r  =  .18,  p  <  .05),  ARMS  ( r  = 
.16,  p  <  .05),  the  number  of  months  since  their  last  Phase  I  training  ( r  =  -.23,  p  <  .01), 
the  average  of  scores  on  the  planning  and  checking  scales  of  the  self-regulation 
survey  (r  =  .16,  p  <  .05),  and  their  most  recent  qualification  score  (r  =  .40,  p  <  .01). 

With  respect  to  the  performance  on  our  assessments,  we  found  that  Marines 
scored  higher  on  our  prior  knowledge  measure  (a)  if  they  had  Phase  I  training  (r  = 
.23,  p  <  .01);  and  (b)  the  more  recently  (in  months)  they  had  Phase  I  training  ( r  =  -.34, 
p  <  .05).  Also,  an  unexpected  negative  relationship  was  found  on  the  shot  group 
depiction  task.  The  higher  Marines  performed  on  the  shot  group  task  (a)  the  lower 
the  number  of  hours  they  reported  spending  shooting  a  rifle  as  part  of  their  USMC 
duties  (r  =  -.16,  p  <  .05);  (b)  the  less  frequently  they  reported  shooting  a  rifle  outside 
their  USMC  duties  (r  =  .16,  p  <  .05);  and  (c)  the  more  recently  (in  months)  they  had 
Phase  I  training  ( r  =  -.25,  p  <  .01).  For  the  knowledge  mapping  measure.  Marines 
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who  scored  higher  on  the  knowledge  map  (a)  reported  their  jobs  as  non-ground 
combat  ( r  =  .20,  p  <  .05);  (b)  had  had  Phase  I  training  (r  =  .19,  p  <  .05);  (c)  reported 
spending  a  lower  number  of  hours  shooting  a  rifle  as  part  of  their  USMC  duties  ( r  = 
-.20,  p  <  .05);  and  (d)  had  more  recently  (in  months)  had  Phase  I  training  (r  =  -.20,  p  < 
.05). 

Finally,  the  best  predictor  of  qualification  scores  was  from  the  following  set  of 
variables:  (a)  average  of  scores  on  the  planning  and  checking  scales,  (b)  number  of 
months  since  the  Marine's  last  Phase  I  training,  (c)  the  prior  knowledge  score,  (d) 
most  recent  qualification  score,  (e)  if  the  Marine  took  the  coach's  course,  (f)  shot 
group  depiction  score,  (g)  ARMS  score,  and  (h)  number  of  strong  causal  links  in 
their  knowledge  map.  The  resulting  multiple  R  for  this  sample  was  .59  (34%  of  the 
variance  in  qualification  scores  explained).  Using  a  regression  model  with  only 
background  variables,  the  multiple  R  was  .49  (24%  of  the  variance  explained).  Thus, 
the  addition  of  cognitively  based  measures  explained  10%  more  variance  in  the 
qualification  score  beyond  what  was  explained  by  background  variables  alone.  This 
increase  suggests  that  there  is  a  cognitive  component. 

Findings 

In  general,  our  very  preliminary  results  suggest  the  following: 

1.  A  cognitive  component  exists  for  rifle  marksmanship.  Evidence  for  this 
interpretation  was  found  in  the  statistically  significant  relationships 
between  our  measures  of  knowledge  and  qualification  scores,  and  in  the 
regression  equation  used  to  predict  qualification  scores. 

2.  The  single  best  predictor  of  qualification  score  was  a  Marine's  most  recent 
qualification  score  (r  =  .40). 

3.  The  second  single  best  predictor  of  qualification  score  was  a  Marine's  score 
on  our  prior  knowledge  measure  (r  =  .31). 

4.  When  we  combined  background  information  and  knowledge  of  rifle 
marksmanship  information,  our  preliminary  regression  model  predicted 
qualification  scores  nearly  as  well  as  the  highest  rifle-simulator-based 
results.  The  highest  correlation  was  reported  by  Hagman  (1998),  with  r  = 
.69;  our  study  found  multiple  R  =  .59. 

5.  Using  only  background  variables  —  most  recent  qualification  score,  how 
long  ago  the  Marine  had  Phase  I  training,  if  the  Marine  took  the  coach's 
course— predicted  qualification  scores  with  a  multiple  R  of  .49. 
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6.  The  inclusion  of  knowledge-based  measures  adds  substantially  to  the 
prediction  of  qualification  scores,  above  and  beyond  background  variables 
alone.  Including  information  about  Marines'  knowledge  of  rifle 
marksmanship  improved  the  predictability  substantially  (multiple  R  =  .59). 


Development  and  Internet  Delivery  of  HUEY  Training  Prototypes 

Researchers  at  Behavioral  Technology  Laboratories  and  CRESST  have 
developed  a  conception  for  the  integration  of  assessment  and  interactive  distributed 
training  that  is  represented  in  Figure  1. 


Assessment  Tools 


Specification  Simulation  Player  & 
Tutorial  Player 


Figure  1.  Battlesight  Zero  lesson  for  Marine  marksmanship. 


In  this  vision,  learners  run  browser  client  programs  that  access  assessment 
tools  that  are  delivered  by  a  central  server.  Based  on  the  detailed  assessments, 
particular  training  modules  are  provided  to  the  student.  In  a  typical  training 
sequence,  a  Marine  would  first  complete  certain  assessments  and  would  then 
interact  with  a  training  module  chosen  to  address  deficits  detected  by  those 
assessments.  Depending  on  performance  during  that  training,  other  assessments 
would  then  be  selected.  The  results  of  these,  in  turn,  would  determine  which 
subsequent  training  modules  would  be  delivered  to  the  student.  This  process  would 
continue  either  until  criterion  assessment  results  were  obtained  or  until  available 
training  materials  were  exhausted.  A  report  of  the  remediation  carried  out  and  any 
remaining  conceptual  deficits  about  marksmanship  for  that  student  would  then  be 
generated. 

One  goal  of  this  project  is  to  produce  a  proof-of-concept  prototype  that 
implements  the  major  elements  of  this  vision  for  integrated  assessment  and 
distributed  training  delivered  on  the  Web. 
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Training  Content  Modules  in  the  Prototype 

Four  major  interactive  Web-delivered  training  modules  are  envisioned.  Two  of 
these  have  been  largely  completed  and  two  are  in  development.  The  completed 
modules  teach  the  Battlesight  Zero  (BZO)  procedure  and  shot  group  analysis.  The 
modules  in  development  will  teach  about  Sight  Picture  and  effective  use  of  the  data 
book. 

Battlesight  Zero.  The  BZO  module  guides  the  Marine  through  the  process  of 
carrying  out  the  BZO  procedure.  In  the  process,  it  may  remediate  misconceptions 
that  it  encounters  during  the  training  process.  The  purpose  of  this  module  is  not  to 
teach  the  sensory  and  motor  skills  that  can  only  be  learned  by  actually  handling  and 
firing  a  weapon.  Its  purpose  is  to  teach  the  steps  of  and  the  reasons  for  the 
Battlesight  Zero  procedure.  In  this  module,  students  practice  creating  shot  groups 
and  deciding  how  to  adjust  the  sights  based  on  those  shot  groups.  Finally,  they  must 
adjust  the  sights  back  to  a  zero  wind  condition  to  complete  the  BZO  process. 

Figure  2  shows  the  beginning  of  the  BZO  module,  in  which  the  student  is 
introduced  to  the  lesson.  It  explains  how  the  student  aims  by  placing  the  sights  on 
the  target  and  that  shots  are  fired  by  pressing  the  space  bar.  In  Figure  3,  the  lesson 
has  noticed  that  the  student  attempted  to  fire  a  shot  while  the  gun  was  not  even 
pointed  at  the  target.  Students  cannot  proceed  unless  they  follow  the  directions. 
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Figure  2.  Introduction  to  Battlesight  Zero. 


Figure  3.  Remediating  gross  aiming  error. 
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Figure  4.  Introduction  to  sight  adjustment. 


Figure  5.  Remediating  use  of  rear  elevation 
adjustment  during  BZO. 


Once  a  group  of  shots  has  been  fired,  the  target  is  taken  down  and  marked,  and 
is  then  raised  up  again.  The  view  then  zooms  in  on  the  marked  target,  as  shown  in 
Figure  4.  The  student  is  instructed  on  how  to  adjust  the  sights  and  is  given  an 
opportunity  to  do  so. 

If  a  student  (wrongly)  attempts  to  use  the  Rear  Elevation  adjustment,  the  lesson 
catches  the  error  and  explains  that  this  adjustment  is  not  used  in  the  BZO  procedure 
(Figure  5).  When  the  Marine's  sight  adjustment  is  finished,  he  or  she  is  given 
feedback  on  the  adjustments,  as  in  Figure  6.  The  student  then  fires  another  group  of 
shots  and  readjusts  the  sights.  After  firing  a  final  group  of  shots  to  see  the  effects  of 
the  final  adjustments,  the  student  is  reminded  that  this  BZO  process  is  an  important 
precondition  to  being  able  to  qualify  (Figure  7). 
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Figure  6.  Providing  feedback  on  sight 
adjustments. 


Figure  7.  Relating  the  BZO  process  to 
qualification. 


Shot  group  analysis.  The  shot  group  analysis  training  module  teaches  the 
Marine  to  classify  four  shot  group  patterns:  good  groups,  trigger  control  problem 
groups,  breathing  problem  groups,  and  groups  with  complex  (or  undiagnosable) 
problems.  It  would  be  possible  to  extend  this  module  to  teach  the  recognition  of 
other  shot  pattern  types,  if  Marine  subject  matter  experts  can  agree  on  other  useful 
characterizations  of  shot  group  patterns. 


Figure  8.  Not  recognizing  trigger  control  Figure  9.  Remediation  with  visualization, 

problem. 


In  Figure  8,  a  Marine  has  failed  to  recognize  that  a  horizontally  dispersed  shot 
pattern  is  a  sign  of  a  trigger  control  problem.  In  addition  to  informing  the  student 
that  that  is  the  depicted  problem,  it  also  brings  up  a  visualization,  in  Figure  9,  that 
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animates  the  rifle  pivoting  around  the  trigger  hand  while  shots  appear,  one  by  one, 
in  a  horizontally  spread-out  pattern. 

Misconceptions  about  breath  control  are  also  remediated  with  a  visualization 
in  which  the  breathing  of  a  stick  figure  soldier  (whose  torso  expands  and  contracts) 
causes  the  rifle  to  pivot  vertically.  This  is  correlated  with  the  vertical  movement  of 
the  sight  over  the  target  (Figure  10). 


Figure  10.  Remediating  failure  to  recognize 
breath  control  problem  with  visualization. 


Sight  picture  module.  Marines  need  to  be  taught  how  to  create  the  proper 
sight  picture,  by  focusing  not  on  the  target  or  on  the  rear  sight,  but  rather  on  the 
front  sight.  Figure  11  shows  a  portion  of  the  in-progress  module  on  this  topic. 
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Figure  11.  Sight  picture  simulation  (in 
development). 


Data  book  usage.  The  fourth  module,  which  is  currently  being  designed,  will 
teach  Marines  how  to  use  the  data  book,  and  will  give  them  evaluated  practice  in 
using  it  to  record  conditions  and  shooting  performance.  CRESST  investigators  have 
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found  that  extensive  data  book  usage  is  highly  correlated  with  good  qualification 
performance. 

Authoring  Tools  to  Promote  the  Maintainability  of  ADL  Content 

Under  ONR  funding,  Behavioral  Technology  Laboratories  is  developing  new 
tools  to  support  the  development,  maintenance,  and  modification  of  ADL  content. 

Chief  among  these  tools  is  iRides  Author,  an  application  that  supports  the 
development  of  modifiable  simulations  and  modifiable  training  control 
specifications.  Figure  12  shows  several  of  the  authoring  interfaces. 

The  modifiability  of  training  content  means  that  it  will  be  possible  for  other 
contractors  or  authorized  Marine  personnel  to  modify  any  aspect  of  the  training 
content.  This  can  include  the  behavior  of  simulated  systems,  the  wording  of 
directives  or  explanations,  and  the  performance  criteria  for  students,  along  with 
many  other  characteristics  of  the  training  package.  It  is  expected  that  this  use  of 
authoring  tools,  rather  than  low-level  programming  languages,  will  make  it  feasible 
for  the  Marine  Corps  to  adapt  the  content  for  future  requirements. 


Interpreter/ 


Figure  12.  Authoring  advanced  content. 
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The  Baseline  Description  of  Existing  Marksmanship  Training 

This  section  summarizes  an  initial  study  of  Marine  Corps  training  undertaken 
as  part  of  the  Office  of  Naval  Research  Capable  Manpower  Future  Naval 
Capabilities  program.  The  study  is  part  of  a  larger  Capable  Manpower  program 
investigating  the  employment  of  advanced  distributed  learning  (ADL)  technology  in 
the  Navy  and  Marine  Corps,  with  the  overall  goal  of  developing  practical  guidelines 
and  procedures  to  support  effective  ADL  employment  in  the  future.  This  larger 
program  is  conducting  a  number  of  qualitative  studies  of  training  curricula  as 
delivered  under  both  traditional  (e.g.,  classroom-based  or  practice-based)  and  ADL 
formats.  These  studies  are  providing  data  on  how  instructional  content  is  being 
delivered  to  Marine  Corps  and  Navy  personnel,  and  on  how  organizational 
processes  adapt  (or  need  to  adapt)  to  make  effective  use  of  ADL-based  delivery. 
Once  completed,  the  data  from  these  studies  will  provide  the  basis  for  guidelines 
and  supporting  case  studies  intended  to  help  the  Navy  (via  NETC)  and  Marine 
Corps  (via  the  DLC)  better  design  and  employ  ADL  technology. 

Researchers  from  CHI  Systems,  Inc.  conducted  this  study  in  two  phases.  The 
first  phase  involved  interviewing  rifle  training  administrators  and  instructors  about 
existing  practices  and  processes  at  a  specific  weapons  training  battalion.  In  the 
second  phase,  a  researcher  spent  two  weeks  at  the  training  battalion,  directly 
participating  in  all  aspects  of  marksmanship  training,  just  as  any  regular  trainee 
would  (and  eventually  qualifying  as  an  Expert).  In  addition  to  the  direct  observation 
of  and  participation  in  the  process  from  a  trainee's  perspective,  the  second  phase 
also  involved  conducting  extensive  interviews  with  many  of  the  participants 
(trainers  and  trainees),  and  documenting  their  experiences  with  the  existing  training 
program,  as  well  as  their  attitudes  toward  and  opinions  of  different  ADL  training 
strategies.  To  anticipate  future  impediments  to  ADL  employment,  trainees  and 
trainers  were  asked  to  discuss  critical  training  processes  and  tasks,  and  to  comment 
specifically  on  the  role(s)  that  the  advanced  technology  (e.g.,  an  existing  digitized 
marksmanship  course  and/or  the  ISMT  device)  should  play,  if  any,  in  a  revamped 
training  program.  During  the  observations  and  interview,  explicit  attention  was 
paid  to  noting  the  subtle  (and  not-so-subtle)  indicators  of  skepticism,  optimism, 
frustration,  enthusiasm,  and  other  important  attitudinal  variables,  which  signal  the 
presence  or  absence  of  organizational  support  for  particular  ADL  initiatives. 
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Findings  Relating  to  Current  Marksmanship  Training 

The  findings  of  the  study  are  summarized  below,  according  to  their  relevance 
to  Phase  I  training,  Phase  II  training,  or  ADL-specific  issues. 

Phase  I.  The  study  found  widespread  dissatisfaction  with  Phase  I  training, 
reflecting  problems  with  the  existing,  conventional  approach,  which  brings  a  large 
group  of  trainees  together  in  a  classroom  for  a  day  or  two  to  listen  to  lectures 
covering  the  mechanics  and  artistry  of  shooting  the  M16A2.  A  number  of  factors 
have  combined  to  undermine  the  effectiveness  of  Phase  I  training  at  the  WTBN.  The 
majority  of  supervisors  at  the  WTBN  cited  the  absence  of  centralized  command  over 
Phase  I  as  one  of  the  biggest  problems— if  not  the  biggest— with  current 
marksmanship  training.  Although  unit  commanders  attest  to  the  fact  that  their 
trainees  have  gone  through  the  full  set  of  lectures  prior  to  arriving  at  the  WTBN  for 
field  practice,  supervisors  at  the  WTBN  strongly  suspect  that  many  trainees  receive 
only  cursory  instruction.  Supervisors  told  accounts  of  shooters  who  appeared  to  be 
struggling  with  even  the  most  basic  procedures  during  Phase  II  and,  when 
questioned  about  their  obvious  ignorance,  admitted  that  they  had  received  little  or 
no  Phase  I  training— despite  assurances  by  their  unit  commanders  to  the  contrary. 
Supervisors,  as  a  result,  often  suggest  (often  based  on  such  accounts  of  inadequate 
preparation  for  field  practice)  that  full  control  over  Phase  I  should  be  returned  to  the 
Training  Command,  in  order  to  make  this  part  of  the  training  cycle  "better  and  more 
verifiable."  At  the  same  time,  however,  they  acknowledge  that  putting  such  a  policy 
into  effect  will  be  an  uphill  battle,  considering  unit  commanders'  reluctance  to  give 
up  their  Marines  for  more  than  a  few  days. 

Even  when  trainees  receive  full  Phase  I  classroom  training,  either  at  their  units 
or  at  a  WTBN,  there  are  problems  in  retention  of  the  material  that  is  presented  to 
them.  Instructors  have  little  time  to  cover  many  complex  topics,  including  the 
internal  workings  of  the  rifle,  the  fundamentals  of  breathing,  aiming  and  trigger 
control,  and  the  mechanics  of  sight  alignment  and  sight  adjustment.  Any  one  of 
these  topics  is  difficult  for  an  instructor  to  convey  and  for  a  trainee  to  assimilate. 
There  is  also  a  general  lack  of  individualized  instruction  (e.g.,  coaching  and 
feedback)  and  a  reliance  on  a  passive  learning  process  (listening  to  rather  than 
engaging  with  the  material).  Many  trainees  report  getting  relatively  little  out  of 
Phase  I.  Other  factors,  including  poor  acoustics,  lack  of  air  conditioning  or  heating, 
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and  dryness  of  content,  also  make  it  difficult  for  trainees  to  pay  attention  to,  and 
thus  learn,  the  fundamental  concepts  and  procedures  presented  in  Phase  I  lectures. 

Phase  II.  Participant  observation  on  the  firing  range  confirmed  just  how 
difficult  it  is  for  shooters  who  have  not  mastered  the  fundamentals  during  Phase  I  to 
use  their  practice  time  effectively  during  the  Phase  II  field  practice.  Without  a  clear 
understanding  of  the  fundamentals,  trainees  cannot  effectively  diagnose  and 
remedy  their  deficiencies  during  the  four  days  of  practice  that  they  are  afforded. 
Trainees  have  particular  difficulty  determining  whether  their  inaccurate  firing 
patterns  are  the  result  of  poor  body  positioning,  incorrect  sight  adjustment,  or  both. 
As  a  result  of  this  confusion,  struggling  shooters  spend  a  significant  portion  of  their 
valuable  practice  time  floundering,  rather  than  systematically  honing  their  skills  for 
qualifying  day.  In  some  cases,  trainees  become  so  frustrated  that  they  attempt  to 
qualify  early,  despite  being  unprepared  to  do  so,  because  the  practice  does  not  seem 
to  be  improving  their  skills.  Another  clear  sign  of  frustration  is  the  refusal  of  many 
of  those  who  fail  to  qualify,  on  their  first  attempt,  to  take  advantage  of  the  additional 
qualification  opportunity  at  the  end  of  Phase  II. 

Coaches  often  provide  critical  assistance  to  struggling  shooters  hoping  to 
qualify  or  to  good  shooters  hoping  to  qualify  as  experts,  but  the  individualized 
instruction  only  benefits  trainees  who  already  have  a  firm  grasp  on  the  basic 
concepts  covered  in  Phase  I.  By  observing  shooters,  coaches  point  out  problems  with 
breathing,  aiming,  and  trigger  pull  that  the  shooters  themselves  would  have 
difficulty  identifying.  Coaches  also  make  helpful  suggestions  about  adjusting  sights 
to  compensate  for  the  effects  of  weather  and  distance.  To  have  a  lasting  effect  on 
shooting  proficiency,  however,  the  help  that  trainees  receive  from  coaches  must  be 
internalized  and  integrated  into  their  own  thought  processes,  so  that  trainees  can  do 
things  like  adjust  windage  and  elevation  settings,  as  well  as  modify  body  positions, 
on  their  own.  Trainees  also  have  to  rely  on  their  own  knowledge  in  order  to  choose 
between,  or  attempt  to  reconcile,  the  conflicting  advice  that  they  sometimes  receive 
from  coaches.  It  is  not  unusual  for  one  coach  to  tell  a  struggling  shooter  one  thing, 
and  then  another  to  come  along  and  tell  him  or  her  the  exact  opposite,  as  coaching 
appears  to  be  as  much  art  as  science.  In  such  situations,  trainees  must  be 
discriminating,  selectively  integrating  the  opinions  of  others  into  their  own  personal 
diagnoses  and  remedial  strategies. 

Only  by  becoming  self-sufficient  can  shooters  take  the  lessons  learned  during 
the  practice  days  (when  coaches  are  available)  and  apply  them  effectively  on 
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qualification  day  (when  they  are  not).  By  way  of  illustration,  a  number  of  trainees 
interviewed  performed  well  during  practice  days,  as  they  worked  closely  with 
coaches,  but  then  struggled  on  qualification  day  because  the  support  systems  or 
"crutches"  (trainees'  word)  upon  which  they  had  relied  were  taken  away  from  them. 
Because  they  had  not  mastered  the  fundamentals,  these  trainees  had  become 
dependent  on  coaches,  and,  thus,  were  unable  to  take  corrective  actions  on  their 
own. 

Findings  Relating  to  ADL  Employment 

An  important  step  toward  improving  the  proficiency  of  shooting  in  the  Marine 
Corps  is  to  develop  a  more  dynamic  and  effective  way  of  covering  the  fundamental 
concepts  in  Phase  I.  ADL  can  play  an  important  role  in  a  revamped  program,  but  to 
do  so,  the  new  method  and  tools  must  be  employed  in  a  way  that  avoids  likely 
pitfalls  and  enhances  potential  benefits. 

Two  forms  of  ADL  technology  were  available  at  the  WTBN  at  the  time  of  the 
study: 

1.  The  MCI  automated  digitized  marksmanship  course,  on  CD-ROM,  and 

2.  The  ISMT  hardware  simulator. 

The  ISMT  had  been  available  for  several  years  at  the  WTBN,  whereas  the 
digitized  course  had  been  delivered  only  a  couple  of  months  prior  to  the 
researcher's  arrival  and  had  not  been  integrated  into  the  training  curriculum. 
Trainers  did  not  have  a  clear  idea  of  how  the  digitized  course  was  intended  to  be 
used  or  what  parts  of  existing  training  it  could  supplement/ complement/ replace. 
These  computer-based  instructional  tools  have  the  potential  to  improve  training,  by 
providing  the  kind  of  intensive,  individualized  instruction,  guidance,  and  feedback 
that  cannot  otherwise  be  provided  during  Phase  I.  The  ISMT  can,  in  theory, 
familiarize  shooters  with  the  M16A2  and  even  help  diagnose  problems  with 
shooting.  The  digitized  marksmanship  course  has  the  potential  to  teach  the 
fundamentals  of  marksmanship  in  an  engaging  way,  through  the  use  of  interactive, 
multimedia  presentations  and  exercises.  However,  the  study  identified  two 
employment  problems  and/or  obstacles  that  must  be  addressed  before  these  ADL 
technologies  can  be  effectively  incorporated  into  the  formal  training  cycle: 

Technology  failures  have  created  a  negative  anticipation.  The  consensus 
opinion  of  those  interviewed,  trainers  and  trainees  alike,  was  that  the  ISMT 
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hardware  simulator  had  never  become  an  integral  part  of  training,  and  had 
gradually  fallen  out  of  favor  and  into  disuse,  due  to  technical  problems  that  have 
plagued  the  instructional  technology.  One  common  complaint  heard  by  the 
researcher  concerned  the  difficulties  that  instructors  and  technicians  have  had 
keeping  the  ISMT  operational.  It  apparently  takes  a  considerable  amount  of  time  for 
a  technician  to  get  the  simulator  working  correctly,  and  even  then,  the  ISMT 
frequently  breaks  down  during  use.  The  researcher  experienced  these  problems 
himself  when  he  repeatedly  attempted,  with  no  success,  to  utilize  the  simulator. 
Some  respondents  went  so  far  as  to  suggest  that  frustrations  and  disappointments 
with  the  ISMT  have  created  a  less  than  hospitable  environment  at  the  Battalion  for 
the  introduction  of  a  new  ADL  product,  such  as  the  digitized  marksmanship  course, 
regardless  of  the  latter's  merits. 

Adequate  technology  access  does  not  yet  exist.  With  only  30  or  so  computers 
at  the  local  Learning  Resource  Center,  and  approximately  10  stations  at  the  ISMT, 
there  is  simply  not  enough  technology  available  at  the  Battalion  to  reach  all  of  the 
trainees  (hundreds)  who  arrive  each  week.  A  program  proposed  by  one  Training 
Command  was  to  target  the  low-scoring  marksman  and  the  shooter  who  totally  fails 
marksmanship  qualification— a  small  subset  of  trainees.  In  order  to  increase  the 
proficiency  of  these  struggling  shooters,  the  remedial  program  would  put  this  select 
group  through  targeted  class  sessions  utilizing  the  digitized  course.  Then,  after 
having  their  knowledge  of  basic  concepts  refreshed,  the  participants  would  receive 
follow-up  lessons  with  the  hardware  simulator,  before  being  given  an  additional 
opportunity  to  qualify.  Although  focusing  on  those  who  performed  poorly  on 
qualification  day  addresses  the  problem  of  accessibility,  this  type  of  remediation 
program  creates  a  problem  of  sequencing.  Selecting  trainees  for  remediation  based 
on  results  during  qualification  means  that  ADL  technologies  are  used  after  Phase  II 
field  practice  has  been  completed.  This  precludes  participants  from  applying,  on  the 
firing  line,  the  lessons  that  they  presumably  learned  through  ADL.  Most  instructors 
and  coaches  interviewed  for  the  study  stated  that  proceeding  directly  from  the 
follow-up  lessons  with  the  ISMT  to  the  additional  qualification  opportunity  would 
be  counterproductive,  since  there  is  limited  transference  of  skills  from  the  hardware 
simulator  to  the  firing  range.  These  experts  recommended  that  participants  spend  at 
least  one  additional  day  on  the  firing  range  before  the  second  attempt  at 
qualification,  so  that  they  can  consolidate  what  they  have  learned  during 
remediation.  This  raises  an  important  question:  Would  unit  commanders  allow  their 
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Marines  to  stay  for  the  additional  day(s)  necessary  to  complete  this  extended 
remedial  training? 

A  distance-learning  mode  of  delivery  can  address  the  issues  of  accessibility  and 
sequencing  without  requiring  additional  investment  in  technology,  at  least  in  the 
case  of  the  digitized  course.  Under  this  employment  strategy,  the  CD-ROM  would 
be  made  available  at  the  individual  units,  so  that  Marines  could  access  the  ADL 
lessons  prior  to  coming  to  the  WTBN.  The  course  would  be  accessed  through  existing 
computers  (personal  or  work),  and  would  either  supplement  or  replace  existing 
Phase  I  training.  Making  the  ISMT  more  accessible  to  trainees,  on  the  other  hand, 
would  require  a  more  significant  investment. 

Next  Steps 

CRESST  will  complete  analysis  of  Study  C  data  and  will  document  results  of  all 
studies  in  a  Final  Report.  BTL  will  document  its  marksmanship  simulations  in  a 
Final  Report.  CHI  will  extend  its  study  of  marksmanship  training  with  a  study  of 
how  the  MCI  Rifle  Marksmanship  online  course  is  being  used,  and  may  conduct 
additional  qualitative  field-based  studies  of  USMC  DL  implementations  to  support 
development  of  practical  guidelines  and  procedures  for  effective  DL  employment. 
The  Final  Reports  and  prototype  assessment  and  simulation  tools  will  be  delivered 
to  ONR,  the  Camp  Lejeune  Weapons  Training  Battalion,  and  the  USMC  Training 
and  Education  Command. 

The  KMT  project  has  produced  prototype  assessment  and  instructional 
simulation  tools  of  potential  benefit  to  USMC  marksmanship  training.  The  ONR 
KMT  funding  is  R&D  money,  however,  and  cannot  support  the  transition  of  these 
tools  from  Science  and  Technology  (S&T)  advanced  technology  prototypes  to  the 
fieldable  tools  needed  for  insertion  in  the  marksmanship  training  process.  If 
transition  money  were  available,  either  from  the  Navy  or  the  Marine  Corps,  the 
following  work  might  be  completed: 

•  Validation  of  the  assessments'  ability  to  go  beyond  prediction  of  overall 
shooting  performance  to  identify  specific  knowledge  gaps,  predict  the 
impact  of  the  missing  knowledge  on  shooting  performance,  and  identify  the 
remediation  required  to  fill  the  gap. 

•  Shrink-wrapping,  bullet-proofing,  and  validating  a  complete  assessment 
and  training  package  based  on  the  prototypes. 
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•  Qualitative  studies  of  the  results  of  fielding  the  assessment  and  training 
package  produced  with  transition  funding,  including  analyses  of  how 
organizational  processes  adapt  (or  need  to  adapt)  to  make  effective  use  of 
DL  delivery,  and  guidelines  and  a  supporting  case  study  helping  the  Navy 
(via  NETC)  and  Marine  Corps  (via  the  DLC)  better  design  and  employ  DL. 

In  addition,  the  following  work  might  be  considered  for  additional  R&D 
funding: 


•  Integrating  assessment  and  training  with  the  forthcoming  Marine  LOMAH 
(Location  Of  Miss  And  Hit)  system.  This  would  make  it  possible  to  assess 
shooting  performance  directly  and  automatically,  and  to  select  appropriate 
training  modules  based  on  that  performance. 

•  Hand-held  assessment  and  training  modules.  These  could  be  used  to  teach 
data  book  activities  in  the  context  of  real  shooting  environments,  with 
automatic  evaluation  and  feedback.  Advice  could  be  provided  on  the  firing 
line,  and  instructors  could  be  automatically  alerted  about  problems.  If 
integrated  with  the  LOMAH  system,  these  hand-held  devices  could  also 
provide  feedback  and  advice  about  actual  shooting  performance. 
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